Search for: All records

What do high school students learn from a two-day datathon during which they tackle data to visualize the impact of biased data on healthcare decisions? How do they interact with their team of high school students, data scientists, clinicians, and teachers? What did we, the developers and leaders of the datathon, learn? How would we approach it differently next year? Our goal is to answer these questions plus share lessons learned. We will then divide the audience into teams to brainstorm ways to approach and solve some of the problems we experienced and hopefully recruit some audience members to participate in our June 2025 Brown University Health Artificial Intelligence (AI) Systems Thinking for Equity (HASTE) Datathon in Providence, Rhode Island (Brown University Datathon, 2024). 

Abstract ObjectiveData extraction from the published literature is the most laborious step in conducting living systematic reviews (LSRs). We aim to build a generalizable, automated data extraction workflow leveraging large language models (LLMs) that mimics the real-world 2-reviewer process. Materials and MethodsA dataset of 10 trials (22 publications) from a published LSR was used, focusing on 23 variables related to trial, population, and outcomes data. The dataset was split into prompt development (n = 5) and held-out test sets (n = 17). GPT-4-turbo and Claude-3-Opus were used for data extraction. Responses from the 2 LLMs were considered concordant if they were the same for a given variable. The discordant responses from each LLM were provided to the other LLM for cross-critique. Accuracy, ie, the total number of correct responses divided by the total number of responses, was computed to assess performance. ResultsIn the prompt development set, 110 (96%) responses were concordant, achieving an accuracy of 0.99 against the gold standard. In the test set, 342 (87%) responses were concordant. The accuracy of the concordant responses was 0.94. The accuracy of the discordant responses was 0.41 for GPT-4-turbo and 0.50 for Claude-3-Opus. Of the 49 discordant responses, 25 (51%) became concordant after cross-critique, increasing accuracy to 0.76. DiscussionConcordant responses by the LLMs are likely to be accurate. In instances of discordant responses, cross-critique can further increase the accuracy. ConclusionLarge language models, when simulated in a collaborative, 2-reviewer workflow, can extract data with reasonable performance, enabling truly “living” systematic reviews.